Gradient-Based Training of Gaussian Mixture Models for High-Dimensional Streaming Data

Abstract We present an approach for efficiently training Gaussian Mixture Model (GMM) by Stochastic Gradient Descent (SGD) with non-stationary, high-dimensional streaming data. Our scheme does not require data-driven parameter initialization (e.g., k-means) and can thus be trained based on a random initial state. Furthermore, the allows mini-batch sizes as low 1, which are typical streaming-data settings. Major problems in such settings undesirable local optima during early phases numerical instabilities due to high data dimensionalities. introduce adaptive annealing procedure address first problem, whereas eliminated exponential-free approximation standard GMM log-likelihood. Experiments variety of visual non-visual benchmarks show that our SGD completely without, instance, k-means centroid initialization. It also compares favorably online variant Expectation-Maximization (EM)—stochastic EM (sEM), it outperforms large margin very